GUIエージェント(Computer use)
AIエージェント
マルチエージェントシステム
Vision Language Model
マルチモーダルモデル
OCR
視覚文書理解
LLMでPCを操作！？Claudeの新機能「computer use」を早速試してみた
https://qiita.com/hedgehog051/items/eddcf26ad5d1dc8086d5
Omniparser
https://huggingface.co/microsoft/OmniParser
OmniParser for pure vision-based GUI agent
https://www.microsoft.com/en-us/research/articles/omniparser-for-pure-vision-based-gui-agent/
Large Language Model-Brained GUI Agents: A Survey
https://arxiv.org/abs/2411.18279
Agent S: An Open Agentic Framework that Uses Computers Like a Human
https://arxiv.org/abs/2410.08164
https://github.com/simular-ai/Agent-S
OmniParser for Pure Vision Based GUI Agent
https://arxiv.org/abs/2408.00203
OS-Atlas: A Foundation Action Model For Generalist GUI Agents
https://github.com/OS-Copilot/OS-Atlas
https://arxiv.org/abs/2410.23218
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
https://arxiv.org/abs/2404.05719
LLM でブラウザを操作する WEB エージェントと周辺技術のざっくり紹介
https://tech.algomatic.jp/entry/survey/agent/web-navigation
BrowserGym
https://github.com/ServiceNow/BrowserGym
browsergym leader board
https://huggingface.co/spaces/ServiceNow/browsergym-leaderboard